The staging channel is archived.

One box, one calendar

Halyard builds freight-booking software for regional brokers, and for most of its life the whole engineering team in Savannah shared a single staging server. Thirty-eight engineers, one box. To put a branch on it you posted in #staging-booking, waited for a thumbs-up from whoever had it, and tried to be done before the next person's slot.

"The channel had more traffic than our incident channel," says Maren Okafor, the engineering lead. "People were scheduling deploys around lunch."

The cost showed up in review time. A pull request that touched rates or dispatch could not be approved until someone had clicked through it on staging, so it sat until staging was free. Halyard's median review was 2.6 days, and Okafor's estimate is that a full day of that was waiting for the box. The worst week came in March, when a half-finished migration from one branch sat on staging over a weekend and three teams spent Monday working out why carrier search returned nothing.

What they changed

Dmitri Vance, a senior engineer on the dispatch team, set Canary up on a Tuesday afternoon in the first week of April. Halyard already ran the app locally from a Compose file, so the first canary.yml was nine lines.

preview:
  from: docker-compose.yml
  services: [web, api, worker]
  database:
    engine: postgres@16
    seed: snapshot://nightly
    mask: [users.email, users.name, carriers.tax_id]
  sleep_after: 30m
  teardown: on_merge

The first attempt did not go well. The Sidekiq worker started before the database had been restored and crashed in a loop, and the preview reported healthy anyway because only web had a health check. Vance added a check to the worker, pushed again, and the second build was live in a little under four minutes.

The runner sits in Halyard's own AWS account, which mattered more than the team expected. Broker contracts forbid shipment data leaving Halyard's infrastructure, and the masking step runs on the runner before a preview ever reads the snapshot. The security review took one meeting.

They did not announce a cutover. For two weeks previews ran alongside staging, and people used whichever was closer to hand. By the third week nobody was booking the box.

What happened

Six weeks after that Tuesday, Okafor archived #staging-booking. The old server was switched off the following Monday.

• Median review dropped from 2.6 days to 1.6.
• Halyard builds about 214 previews in a typical month, one for every pull request.
• Support and the two account managers open preview links themselves, to check a fix before it ships to the broker who reported it.

The change Okafor mentions first is quieter. When something breaks in a preview, it is one pull request with one author, and the question of who broke it does not come up. "Nobody has asked me that since April," she says. "I used to get it twice a week."

Halyard pays by the preview-hour, and because previews sleep after thirty idle minutes the bill has stayed under what the staging server cost to keep running. Next on the list is pointing the nightly end-to-end suite at a preview of main, instead of a machine that somebody has to remember to reset.